A. INTRODUCTION
Kirkpatrick's four levels are designed as a sequence of ways to evaluate training programs. Many practitioners believe that as you proceed through each of the levels, the evaluation becomes more difficult and requires more time. In practice, then, it is common for trainers to get stuck in Levels 1 and 2 and never proceed to Levels 3 and 4, where the most useful data exist. Today, Kirkpatrick certified facilitators stress "starting with the end in mind," essentially beginning with Level 4 and moving backwards in order to better establish the desired outcome before ever planning the training program. When done strategically, reaching these levels does not have to be any more expensive or time consuming, but will still help to ensure on-the-job performance of learned behaviors and skills. The four levels of Kirkpatrick's evaluation model are as follows:
Reaction - what participants thought and felt about the training (satisfaction; "smile sheets")
Learning - the resulting increase in knowledge and/or skills, and change in attitudes. This evaluation occurs during the training in the form of either a knowledge demonstration or test.
Behavior - transfer of knowledge, skills, and/or attitudes from classroom to the job (change in job behavior due to training program). This evaluation occurs 3?6 months post training while the trainee is performing the job. Evaluation usually occurs through observation.
Results - the final results that occurred because of attendance and participation in a training program (can be monetary, performance-based, etc.)
C. INFORMATION
Return on expectations (ROE) demonstrates the degree to which training initiatives satisfy the expectations of key business stakeholders. Return on expectations (ROE) is a foolproof way to show the value of training in the terms desired by key stakeholders. ROE demonstrates the degree to which training initiatives satisfy the expectations of key business stakeholders. Assumptions that may assist training professionals include:
Key stakeholders are high-level managers or executives.
Stakeholder expectations primarily include the accomplishment of the organization's highest-level goals and mission.
Fundamentals:
1. The end is the beginning. Effective training evaluation begins before the program even starts. "Trainers must begin with desired results and then determine what behavior is needed to accomplish them. Then trainers must determine the attitudes, knowledge, and skills that are necessary to bring about the desired behavior(s). The final challenge is to present the training program in a way that enables the participants not only to learn what they need to know but also to react favorably to the program." It is important that the results are defined in measurable terms so that all involved can see the ultimate destination of the initiative. Clearly defined results will increase the likelihood that resources will be used most effectively and efficiently to accomplish the mission. Attempting to apply the four levels after a program has been developed and delivered makes it difficult, if not impossible, to create significant training value. All four levels need to be considered at every step in the program design, execution and measurement.
2. Return on expectations (ROE) is the ultimate indicator of value. Then executives ask for new training, many learning professionals retreat to their departments and start designing and developing suitable programs. While a cursory needs assessment may be conducted, it is rarely taken to a point that completely clarifies expectations of the training contribution to an overall business initiative. Stakeholder expectations define the value that training professionals are responsible for delivering. Learning professionals must ask the stakeholders questions to clarify and refine their expectations on all four Kirkpatrick levels, starting with Level 4 Results. This is a negotiation process in which the training professional makes sure that the expectations are satisfying to the stakeholder and realistic to achieve with the resources available. Once stakeholder expectations are clear, learning professionals then need to convert those typically general wants into observable, measurable success outcomes by asking the question, "What will success look like to you?" Those outcomes then become the Level 4 Results -- the targets to which you can sharply focus your collective efforts to accomplish return on expectations.
3. Business partnership is necessary to bring about positive ROE. Research has validated that training events in and of them selves typically produce about 15% on-the-job application. To increase application and therefore program results, additional actions need to occur before and after formal training. Historically, the role of learning professionals has been Levels 1 and 2, or just the training event. Not surprisingly, this is why many learning professionals spend almost all of their time there. The production of ROE, however, requires a strong Level 3 execution plan. Therefore, it is critical not only to call upon business partners to help identify what success will look like, but also to design a cooperative effort throughout the learning and performance processes in order to maximize results. Before training, learning professionals need to partner with supervisors and managers to prepare participants for training. Even more critical is the role of the supervisor or manager after the training. They are the key people to reinforce newly learned knowledge and skills through support and accountability. The degree to which this reinforcement and coaching happens directly correlates to improved performance and positive outcomes.
4. Value must be created before it can be demonstrated. Research suggests that as much as 90% of training resources are spent on the design, development, and delivery of training events that yield 15% on-the-job application (Brinkerhoff, 2006). Reinforcement that occurs after the training event produces the highest level of learning effectiveness, followed by activities that occur before the learning event. Currently, learning professionals are putting most of their resources into the part of the training process that produces the lowest level of business results. They are spending relatively little time on the pre-training and follow-up activities that translate into the positive behavior change and subsequent results (Levels 3 and 4) that organizations seek. Formal training is the foundation of performance and results. To create ultimate value and ROE, however, strong attention must be given to Level 3 activities. To create maximum value within their organizations, it is therefore essential that learning professionals redefine their roles and extend their expertise, involvement and influence into Levels 3 and 4.
5. A compelling chain of evidence demonstrates your bottom-line value. The training industry is on trial, accused by business leaders of consuming resources in excess of the value delivered to the organization. Following the Kirkpatrick Foundational Principles and using the four levels will create a chain of evidence showing the business value of the entire business partnership effort. It consists of quantitative and qualitative data that sequentially connect the four levels and show the ultimate contribution of learning and reinforcement to the business. When workplace-learning professionals work in concert with their key business partners, this chain of evidence supports the partnership effort and shows the business value of working as a team to accomplish the overall mission. The chain of evidence serves to unify the learning and business functions, not to isolate training or set it apart. This unity is critical for Level 3 execution, where business value is produced. As an example, please view the Kirkpatrick Partners chain of evidence, which shows the value that over 2000 Kirkpatrick certified professionals are bringing to their organizations by applying what they learn in the Kirkpatrick
Four Levels� Evaluation Certification Program.
Information Sheet 2 : Different training/learning evaluation models
A. INTRODUCTION
Evaluation is a state of mind, rather than a set of techniques or specific procedures. Each of these below models outlines an approach to be used to design and create an effective evaluation program. In addition to these models, it is important to understand how evaluation fits into the larger training development models like the Navy's PADDIE+M (Planning, Analysis, Design, Develop, Implement and Evaluate plus Maintenance) and how Blooms Taxonomy (The committee identified three domains of educational activities or learning (Bloom, 1956): Cognitive: mental skills (Knowledge), Affective: growth in feelings or emotional areas (Attitude or self) and Psychomotor: manual or physical skills (Skills)).
C. INFORMATION
Evaluation of workplace learning and training There have been many surveys on the use of evaluation in training and development. While surveys might initially appear heartening, suggesting that many trainers/organizations use training evaluation extensively, when more specific and penetrating questions are asked, it if often the case that many professional trainers and training departments are found to use only 'reactionnaires' (general vague feedback forms for Level I and III surveys), including the invidious 'Happy Sheet' relying on questions such as 'How good did you feel the trainer was?', 'How enjoyable was the training course?', 'How much knowledge did they retain training', and "How effective was computer based training they received?'
As Kirkpatrick, among others, teaches us, even well produced reactionnaires do not constitute proper validation or evaluation of training. For effective training and learning evaluation, the principal questions should be:
To what extent were the identified training needs objectives achieved by the program?
To what extent were the learners' objectives achieved?
What specifically did the learners learn or be usefully reminded of?
What commitment have the learners made about the learning they are going to implement on their return to work?
And back at work,
How successful were the trainees in implementing their action plans?
To what extent did their line managers support them in this?
To what extent has the action listed above achieved a Return on Expectations/Investment (ROE/ROI) for the organization, either in terms of identified objectives satisfaction?
Organizations commonly fail to perform these evaluation processes, especially where:
The requirements sponsor and learning centers, do not have sufficient time to do so, and/or
The requirements sponsor and learning centers does not have sufficient resources - people and money - to do so. Obviously the evaluation must be based on available resources (and the culture atmosphere), which tend to vary substantially from one organization to another. The fact remains that good methodical evaluation produces a good reliable data; conversely, where little evaluation is performed, little is ever known about the effectiveness of the training.
Some of the most common evaluation models (other than Kirkpatrick ROE):
Expanding Kirkpatrick with Phillips Return on Investment (ROI) and Hamblin Model (others).
Learning Styles theory is extremely relevant to training and teaching, and features in Kolb's model, and in the VAK learning styles model (also including a free self-test tool). Learning Styles theory also relates to methods of assessment and evaluation, in which inappropriate testing can severely skew results. Testing, as well as delivery, must take account of people's learning styles, for example some people find it very difficult to prove their competence in a written test, but can show remarkable competence when asked to give a physical demonstration. Text-based evaluation tools are not the best way to assess everybody. Kolb's model Learning Styles theory is extremely relevant to training and teaching, and features in Kolb's model, and in the VAK learning styles model (also including a free self-test tool). Learning Styles theory also relates to methods of assessment and evaluation, in which inappropriate testing can severely skew results. Testing, as well as delivery, must take account of people's learning styles, for example some people find it very difficult to prove their competence in a written test, but can show remarkable competence when asked to give a physical demonstration. Text-based evaluation tools are not the best way to assess everybody. VAK learning styles model - http://www.businessballs.com/kolblearningstyles.htm
The Conscious Competence learning stages theory is also a helpful perspective for learners and teachers. The model helps explain the process of learning to trainers and to learners, and is also helps to refine judgments about competence, since competence is rarely a simple question of 'can or cannot'. The Conscious Competence model particularly provides encouragement to teachers and learners when feelings of frustration arise due to apparent lack of progress. Progress is not always easy to see, but can often be happening nevertheless. Conscious Competence learning stages theory - http://www.businessballs.com/consciouscompetencelearningmodel.htm
Learning Maturity Model - The advent of the Capability Maturity Model (CMM) and People CMM (PCMM) heralded a new dawn in the mindset of stoic into India Inc. The relevance and prestige of support departments—Quality and Human Resources—reached new heights. Quality was no longer a post-mortem back-office activity, but a proactive contributor to external customer satisfaction; the HR manager was no more just a recruiting-and-payroll-processing shadow lurking in the corporate ivory tower, but an enabler of internal customer satisfaction. The best practices of CMM and PCMM organizations can be extrapolated to another vital, but least understood, field—Corporate Learning & Development (L&D). This Learning Maturity Model aims to enable learning organizations systematically graduate in L&D. http://leanlearning.wikispaces.com/learning_maturity_model
Kirkpatrick's four levels are designed as a sequence of ways to evaluate training programs. Many practitioners believe that as you proceed through each of the levels, the evaluation becomes more difficult and requires more time. In practice, then, it is common for trainers to get stuck in Levels 1 and 2 and never proceed to Levels 3 and 4, where the most useful data exist. Today, Kirkpatrick certified facilitators stress "starting with the end in mind," essentially beginning with Level 4 and moving backwards in order to better establish the desired outcome before ever planning the training program. When done strategically, reaching these levels does not have to be any more expensive or time consuming, but will still help to ensure on-the-job performance of learned behaviors and skills. The four levels of Kirkpatrick's evaluation model are as follows:
B. REFERENCES
C. INFORMATION
Return on expectations (ROE) demonstrates the degree to which training initiatives satisfy the expectations of key business stakeholders. Return on expectations (ROE) is a foolproof way to show the value of training in the terms desired by key stakeholders. ROE demonstrates the degree to which training initiatives satisfy the expectations of key business stakeholders. Assumptions that may assist training professionals include:
Fundamentals:
1. The end is the beginning. Effective training evaluation begins before the program even starts. "Trainers must begin with desired results and then determine what behavior is needed to accomplish them. Then trainers must determine the attitudes, knowledge, and skills that are necessary to bring about the desired behavior(s). The final challenge is to present the training program in a way that enables the participants not only to learn what they need to know but also to react favorably to the program." It is important that the results are defined in measurable terms so that all involved can see the ultimate destination of the initiative. Clearly defined results will increase the likelihood that resources will be used most effectively and efficiently to accomplish the mission. Attempting to apply the four levels after a program has been developed and delivered makes it difficult, if not impossible, to create significant training value. All four levels need to be considered at every step in the program design, execution and measurement.
2. Return on expectations (ROE) is the ultimate indicator of value. Then executives ask for new training, many learning professionals retreat to their departments and start designing and developing suitable programs. While a cursory needs assessment may be conducted, it is rarely taken to a point that completely clarifies expectations of the training contribution to an overall business initiative. Stakeholder expectations define the value that training professionals are responsible for delivering. Learning professionals must ask the stakeholders questions to clarify and refine their expectations on all four Kirkpatrick levels, starting with Level 4 Results. This is a negotiation process in which the training professional makes sure that the expectations are satisfying to the stakeholder and realistic to achieve with the resources available. Once stakeholder expectations are clear, learning professionals then need to convert those typically general wants into observable, measurable success outcomes by asking the question, "What will success look like to you?" Those outcomes then become the Level 4 Results -- the targets to which you can sharply focus your collective efforts to accomplish return on expectations.
3. Business partnership is necessary to bring about positive ROE. Research has validated that training events in and of them selves typically produce about 15% on-the-job application. To increase application and therefore program results, additional actions need to occur before and after formal training. Historically, the role of learning professionals has been Levels 1 and 2, or just the training event. Not surprisingly, this is why many learning professionals spend almost all of their time there. The production of ROE, however, requires a strong Level 3 execution plan. Therefore, it is critical not only to call upon business partners to help identify what success will look like, but also to design a cooperative effort throughout the learning and performance processes in order to maximize results. Before training, learning professionals need to partner with supervisors and managers to prepare participants for training. Even more critical is the role of the supervisor or manager after the training. They are the key people to reinforce newly learned knowledge and skills through support and accountability. The degree to which this reinforcement and coaching happens directly correlates to improved performance and positive outcomes.
4. Value must be created before it can be demonstrated. Research suggests that as much as 90% of training resources are spent on the design, development, and delivery of training events that yield 15% on-the-job application (Brinkerhoff, 2006). Reinforcement that occurs after the training event produces the highest level of learning effectiveness, followed by activities that occur before the learning event. Currently, learning professionals are putting most of their resources into the part of the training process that produces the lowest level of business results. They are spending relatively little time on the pre-training and follow-up activities that translate into the positive behavior change and subsequent results (Levels 3 and 4) that organizations seek. Formal training is the foundation of performance and results. To create ultimate value and ROE, however, strong attention must be given to Level 3 activities. To create maximum value within their organizations, it is therefore essential that learning professionals redefine their roles and extend their expertise, involvement and influence into Levels 3 and 4.
5. A compelling chain of evidence demonstrates your bottom-line value. The training industry is on trial, accused by business leaders of consuming resources in excess of the value delivered to the organization. Following the Kirkpatrick Foundational Principles and using the four levels will create a chain of evidence showing the business value of the entire business partnership effort. It consists of quantitative and qualitative data that sequentially connect the four levels and show the ultimate contribution of learning and reinforcement to the business. When workplace-learning professionals work in concert with their key business partners, this chain of evidence supports the partnership effort and shows the business value of working as a team to accomplish the overall mission. The chain of evidence serves to unify the learning and business functions, not to isolate training or set it apart. This unity is critical for Level 3 execution, where business value is produced. As an example, please view the Kirkpatrick Partners chain of evidence, which shows the value that over 2000 Kirkpatrick certified professionals are bringing to their organizations by applying what they learn in the Kirkpatrick
Four Levels� Evaluation Certification Program.
Information Sheet 2 : Different training/learning evaluation models
A. INTRODUCTION
Evaluation is a state of mind, rather than a set of techniques or specific procedures. Each of these below models outlines an approach to be used to design and create an effective evaluation program. In addition to these models, it is important to understand how evaluation fits into the larger training development models like the Navy's PADDIE+M (Planning, Analysis, Design, Develop, Implement and Evaluate plus Maintenance) and how Blooms Taxonomy (The committee identified three domains of educational activities or learning (Bloom, 1956): Cognitive: mental skills (Knowledge), Affective: growth in feelings or emotional areas (Attitude or self) and Psychomotor: manual or physical skills (Skills)).
B. REFERENCES
1. http://www.businessballs.com/trainingprogramevaluation.htm, training programme evaluation (businessballs.com)
C. INFORMATION
Evaluation of workplace learning and training There have been many surveys on the use of evaluation in training and development. While surveys might initially appear heartening, suggesting that many trainers/organizations use training evaluation extensively, when more specific and penetrating questions are asked, it if often the case that many professional trainers and training departments are found to use only 'reactionnaires' (general vague feedback forms for Level I and III surveys), including the invidious 'Happy Sheet' relying on questions such as 'How good did you feel the trainer was?', 'How enjoyable was the training course?', 'How much knowledge did they retain training', and "How effective was computer based training they received?'
As Kirkpatrick, among others, teaches us, even well produced reactionnaires do not constitute proper validation or evaluation of training. For effective training and learning evaluation, the principal questions should be:
And back at work,
Organizations commonly fail to perform these evaluation processes, especially where:
Some of the most common evaluation models (other than Kirkpatrick ROE):
- Expanding Kirkpatrick with Phillips Return on Investment (ROI) and Hamblin Model (others).
- http://dalemunson.com/products/Kirkpatrick_2_Jack Phillips_ROI.pps
- http://ci484-learning-technologies.wikispaces.com/Kirkpatrick's+4+Levels+of+Training
- http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html
- Erik Erikson's Psychosocial (Life Stages) Theory - http://www.businessballs.com/erik_erikson_psychosocial_theory.htm
- Multiple Intelligence theory - http://www.businessballs.com/howardgardnermultipleintelligences.htm
- Learning Styles theory is extremely relevant to training and teaching, and features in Kolb's model, and in the VAK learning styles model (also including a free self-test tool). Learning Styles theory also relates to methods of assessment and evaluation, in which inappropriate testing can severely skew results. Testing, as well as delivery, must take account of people's learning styles, for example some people find it very difficult to prove their competence in a written test, but can show remarkable competence when asked to give a physical demonstration. Text-based evaluation tools are not the best way to assess everybody. Kolb's model Learning Styles theory is extremely relevant to training and teaching, and features in Kolb's model, and in the VAK learning styles model (also including a free self-test tool). Learning Styles theory also relates to methods of assessment and evaluation, in which inappropriate testing can severely skew results. Testing, as well as delivery, must take account of people's learning styles, for example some people find it very difficult to prove their competence in a written test, but can show remarkable competence when asked to give a physical demonstration. Text-based evaluation tools are not the best way to assess everybody. VAK learning styles model - http://www.businessballs.com/kolblearningstyles.htm
- The Conscious Competence learning stages theory is also a helpful perspective for learners and teachers. The model helps explain the process of learning to trainers and to learners, and is also helps to refine judgments about competence, since competence is rarely a simple question of 'can or cannot'. The Conscious Competence model particularly provides encouragement to teachers and learners when feelings of frustration arise due to apparent lack of progress. Progress is not always easy to see, but can often be happening nevertheless. Conscious Competence learning stages theory - http://www.businessballs.com/consciouscompetencelearningmodel.htm
- Learning Maturity Model - The advent of the Capability Maturity Model (CMM) and People CMM (PCMM) heralded a new dawn in the mindset of stoic into India Inc. The relevance and prestige of support departments—Quality and Human Resources—reached new heights. Quality was no longer a post-mortem back-office activity, but a proactive contributor to external customer satisfaction; the HR manager was no more just a recruiting-and-payroll-processing shadow lurking in the corporate ivory tower, but an enabler of internal customer satisfaction. The best practices of CMM and PCMM organizations can be extrapolated to another vital, but least understood, field—Corporate Learning & Development (L&D). This Learning Maturity Model aims to enable learning organizations systematically graduate in L&D. http://leanlearning.wikispaces.com/learning_maturity_model
Additional Study